Demonstrate an understanding of literature exploration and ethical considerations in conducting research.
Lectures 1 and 2 covered why a review matters and which method to use. This lecture covers the four intellectual operations performed once the papers are actually on the desk.
Four operations, four directions of attention: appraisal looks inward at each study, synthesis looks outward across the corpus, gap identification looks at what is absent, and framing supplies the lens. Students who hold these four directions can reconstruct the rest from first principles.
The single highest-yield item is the gap-statement grammar in Section 5.2. Practise writing five, across different CS sub-fields, until you can produce one in ninety seconds.
1. Learning Objectives
- Critically appraise a research paper using structured checklists, evaluating methodological rigour, validity and contribution.
- Synthesise findings from multiple primary studies using at least two formal synthesis techniques.
- Articulate research gaps precisely using the canonical gap-statement grammar.
- Construct a theoretical framework by linking a chosen theory to research questions, propositions and methodological choices.
- Integrate the four techniques into a coherent literature review chapter or related-work section.
Prerequisites: familiarity with basic research designs (experimental, quasi-experimental, case study, survey); literature search strategies and database operators; and working knowledge of IEEE, ACM and APA citation styles.
2. Key Concepts
| Concept | One-line definition |
|---|---|
| Critical appraisal | A structured evaluation of the trustworthiness and value of a research study |
| Synthesis | The integrative recombination of evidence across studies to produce new understanding |
| Research gap | An explicit, defensible absence in the body of knowledge that warrants new research |
| Theoretical framework | The conceptual scaffolding of theory guiding research questions, hypotheses and interpretation |
The four are recursive and overlapping, not sequential silos: synthesis exposes gaps, gaps motivate the choice of theory, and theory redirects what you appraise.
3. Critical Appraisal
3.1 Definition and Purpose
Critical appraisal is the disciplined act of asking not only what a paper claims but whether those claims are warranted by the evidence presented. It is the methodological conscience of the literature review; without it, a literature survey degenerates into an uncritical catalogue.
A researcher must rank claims by the strength of the methods that produced them. In computer science this is subtle: the strongest evidence is sometimes a correctness proof or a benchmark on standardised datasets, whereas in health research it is the randomised controlled trial. Appraisal frameworks must therefore be adapted to the discipline rather than imported wholesale.
3.2 The Three-Lens Model
- Methodological lens. Were the methods appropriate, rigorously executed and transparently reported?
- Conceptual lens. Is the theoretical grounding coherent, and are the constructs well defined?
- Contribution lens. Does the work advance knowledge beyond incremental description?
For each lens, record both strengths and weaknesses, and tag the paper with an appraisal summary — for example, “methodologically sound but conceptually narrow”. Forcing a one-line verdict is what converts passive reading into appraisal.
3.3 Frameworks and Checklists
CASP (Critical Appraisal Skills Programme) is widely used for qualitative studies, with variants for randomised trials, cohort studies and systematic reviews. The qualitative checklist poses ten questions:
- Was there a clear statement of the aims of the research?
- Is a qualitative methodology appropriate?
- Was the research design appropriate to address the aims?
- Was the recruitment strategy appropriate?
- Was the data collected in a way that addressed the research issue?
- Has the relationship between researcher and participants been adequately considered?
- Have ethical issues been taken into consideration?
- Was the data analysis sufficiently rigorous?
- Is there a clear statement of findings?
- How valuable is the research?
AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews, version 2) evaluates sixteen domains, including protocol registration, comprehensiveness of the search strategy, justification for excluding studies, risk-of-bias assessment and consideration of publication bias. Use it when appraising reviews, not primary studies.
- Are the datasets, hyperparameters and random seeds reported?
- Is the experimental setup reproducible — code, environment, container?
- Are baselines appropriate and recent?
- Are statistical tests reported with effect sizes and confidence intervals?
- Is there a threats-to-validity section?
These items now reflect community practice: artefact-evaluation tracks at ICSE, NeurIPS and similar venues enforce several of them.
3.4 Threats to Validity
Cook and Campbell's classical framework distinguishes four validity types, operationalised here for computing research.
| Validity type | Question posed | Typical threat in CS research |
|---|---|---|
| Internal validity | Did the treatment actually cause the observed effect? | Confounding variables, hyperparameter leakage, data snooping |
| Construct validity | Do the operationalisations measure the intended constructs? | Using accuracy as a proxy for “fairness” without conceptual justification |
| External validity | Can the findings generalise? | Single-dataset evaluation, lack of demographic diversity |
| Statistical conclusion validity | Are the statistical inferences correct? | Multiple comparisons without correction, underpowered studies |
Examiners reward precision. Use internal, construct, external and statistical conclusion validity by name — never the generic word “reliable”.
3.5 Worked Appraisal
| Lens | Question | Finding |
|---|---|---|
| Methodological | Were datasets representative? | Single dataset, no temporal split, possible data leakage |
| Methodological | Were baselines compared? | Only logistic regression; no recent transformer-based detectors |
| Conceptual | Is “phishing” well defined? | Definition varies across sources; a narrow URL-only view is adopted |
| Contribution | Does 99.2% advance the field? | Without a proper evaluation protocol the headline metric is misleading |
Verdict: promising idea, but the evidence does not yet support the claimed performance level; treat as preliminary.
The transferable lesson: a high reported number is a claim about an evaluation protocol, not about the world. Phishing URLs are non-stationary, so a random split leaks future information into training — a failure mode that generalises to fraud, intrusion and recommendation research.
4. Synthesis
4.1 Definition and Purpose
Synthesis is the deliberate recombination of evidence from multiple sources into a coherent, often new, understanding. Without synthesis, a literature review is a stack of summaries; with synthesis, it becomes an argument.
4.2 Five Types of Synthesis
- Narrative synthesis — textual integration grouping studies by theme, method or chronology. Flexible and discursive, but susceptible to reviewer bias.
- Meta-analysis — statistical aggregation of effect sizes across quantitative studies. The most rigorous form, but requires comparable metrics.
- Thematic synthesis — codes and themes developed inductively or deductively, then mapped across studies. Originating in health research (Thomas and Harden, 2008), increasingly used in software engineering.
- Integrative synthesis — combines empirical and theoretical sources; suits conceptual papers and design-science research.
- Framework synthesis — uses a pre-existing theoretical or conceptual framework as a scaffold for organising evidence.
Selection logic: comparable quantitative metrics point to meta-analysis; heterogeneous qualitative or mixed evidence points to thematic synthesis; an available theory points to framework synthesis.
4.3 The Synthesis Workflow
- Tabulate the corpus using an extraction sheet: authors, year, method, sample or dataset, findings, limitations.
- Code each study — open coding for inductive synthesis, deductive coding for framework synthesis.
- Cluster codes into themes or categories.
- Relate clusters back to the research questions, identifying convergences and divergences.
- Narrate the relationships, citing studies both as evidence and as counter-evidence.
A conceptual schema for what is happening:
where S is the synthesised claim and Ci are coded observations. In practice f is a defensible inferential move — aggregation, contrast or explanation — not a mechanical combination. Naming the move is what makes the synthesis auditable.
4.4 Worked Example: Summary versus Synthesis
Five studies on software vulnerability prediction using machine learning:
- Study A: Random Forest outperforms SVM on the NVD corpus.
- Study B: deep models excel only with large, balanced corpora.
- Study C: class imbalance remains the dominant performance barrier.
- Study D: static features alone are insufficient; semantic features improve recall.
- Study E: cross-project prediction is poor owing to distributional shift.
“Study A found…, Study B found…, Study C found…” Five sentences, zero claims.
Across the corpus, prediction accuracy is shaped more strongly by data characteristics — imbalance, distributional shift — than by algorithmic choice. While deep models show potential in data-rich settings, classical models retain a Pareto advantage when labelled data is scarce.
This qualifies as synthesis because the claim appears in none of the five studies. It is produced by relating B, C and E (data effects) against A and D (model and feature effects) — a claim greater than the sum of its parts.
5. Gap Identification
5.1 A Taxonomy of Gaps
A research gap is a defensible absence — of evidence, theory, method, population, context or data — that justifies new research. Gap identification is the bridge between the literature review and the research question.
Adapted from Müller-Bloch and Kranz (2015) and refined for computing contexts:
| Gap type | What is absent | What it obliges you to build |
|---|---|---|
| Empirical gap | No study, or insufficient study, of the phenomenon | A study that measures it |
| Theoretical gap | Existing theory does not explain observed patterns, or none applied | A framework, or an application of one |
| Methodological gap | Prior work relies on a narrow set of methods | An alternative method, tool or metric |
| Population / contextual gap | Findings restricted to a demographic, geography or domain | Evaluation in the missing context |
| Data gap | Required datasets do not exist, are inaccessible, or lack key attributes | A dataset contribution |
| Inconsistency gap | Existing studies report conflicting results, unreconciled | A replication or moderator analysis |
- “No study has evaluated the algorithm on low-resource Indic languages.” → population / contextual gap
- “Two systematic reviews report contradictory effect sizes for the same intervention.” → inconsistency gap
- “Technology-adoption theory has not been applied to quantum-computing-as-a-service.” → theoretical gap
5.2 The Canonical Gap-Statement Grammar
Despite [extensive / some] research on [A] in [context C], [little / no] attention has been paid to [B], particularly with respect to [specific dimension]. This is consequential because [reason]. Therefore, this study investigates [research question] using [method] in [context].
- Despite clause — signals awareness of and respect for prior work.
- Gap clause — names the absence precisely.
- Consequence clause — demonstrates significance.
- Therefore clause — positions the new research.
5.3 Four Diagnostic Tests
| Test | Question |
|---|---|
| Evidence test | Can the absence be demonstrated by a structured search? |
| Novelty test | Is the gap distinct from gaps already addressed in recent literature? |
| Tractability test | Can the gap be addressed with available resources, data and time? |
| Significance test | Will closing the gap matter to theory, practice, or both? |
The tractability test is the one students skip and supervisors enforce: a perfectly novel gap that requires inaccessible clinical data is not a viable project.
5.4 Worked Example
The literature reveals that most studies use the ImageNet benchmark; most evaluate only L∞-bounded perturbations; real-world threat models such as physical-world stickers and audio adversarial examples are sparsely studied; and few studies evaluate robustness across demographic subgroups.
Despite extensive research on adversarial robustness in image classifiers, limited attention has been paid to the intersection of robustness and fairness, particularly for L2-bounded perturbations under realistic threat models. This intersection is consequential because deployed classifiers may fail unequally across subgroups, raising both ethical and operational concerns. Therefore, this study investigates the fairness–robustness trade-off in medical imaging classifiers using a constrained optimisation framework.
Every specific phrase — L2, realistic threat models, subgroups — is earned by a documented feature of the literature rather than invented.
6. Theoretical Framing
6.1 Definition and Purpose
Theoretical framing is the act of selecting, adapting and articulating a theory, or combination of theories, that informs every subsequent methodological and interpretive choice. A theoretical framework is not decoration; it is the organising logic of the research.
6.2 Theoretical versus Conceptual Framework
| Aspect | Theoretical framework | Conceptual framework |
|---|---|---|
| Source | Established, citable theory (TAM, DOI) | Researcher-built diagram linking variables |
| Function | Provides explanatory logic and propositions | Clarifies relationships to be tested |
| Testability | Generates hypotheses that can be falsified | Provides a roadmap for analysis |
| Example | Technology Acceptance Model | A diagram linking code-review frequency, defect density and team size |
In practice a theoretical framework can incorporate a conceptual framework when the constructs derive from a theory but are arranged to suit the specific study. A self-drawn variable diagram is acceptable provided it is labelled honestly as such.
6.3 Theories That Recur in CS Research
- TAM (Technology Acceptance Model) — predicts adoption through perceived usefulness and perceived ease of use.
- UTAUT — extends TAM with social and facilitating conditions.
- DOI (Diffusion of Innovations) — explains how innovations spread through social systems.
- TOE (Technology–Organization–Environment) — used in information-systems adoption studies.
- Activity Theory — used in HCI and CSCW to analyse mediated action.
- Constructivism / Situated Learning — used in educational technology and CS pedagogy.
- General Deterrence Theory — used in cybersecurity compliance research.
- PDCA and variants — used in software process improvement research.
- Resource-Based View — explains firm-level IT capabilities.
6.4 Building a Framework in Four Steps
- Identify candidate theories. Surface three to five candidates by reviewing seminal works, systematic reviews and citation tracing. Competition between candidates is what makes the final choice defensible.
- Map constructs to research questions. Verify that the theory's constructs align with the questions; modify or extend where they do not.
- Derive propositions or hypotheses. Translate theoretical relationships into testable statements — for example H1: β1 > 0 for the effect of perceived usefulness on adoption intention.
- Articulate assumptions. State the boundary conditions under which the theory is believed to hold, so that later disconfirmation is interpretable rather than fatal.
- Theory selected: DOI combined with TOE.
- Constructs mapped: relative advantage, compatibility, complexity (DOI); firm size, top-management support, regulatory environment, competitive pressure (TOE).
- H1: relative advantage has a positive effect on serverless adoption.
- H2: top-management support moderates the effect of complexity on adoption.
- Assumptions: startups operate under resource scarcity; respondents are technically literate.
The framework now anchors every subsequent choice — survey instrument, sampling, analysis (partial least squares structural equation modelling) and interpretation. This is precisely why framing precedes design.
7. Integrating the Four Techniques
A robust literature review chapter exhibits the following movement:
- Critical appraisal, per study.
- Extraction and coding.
- Synthesis, across studies.
- Gap identification, within and across themes.
- Theoretical framing, as the lens for interpretation.
- Research questions, anchored in both the gap and the theory.
A common failure is to perform critical appraisal and synthesis but skip theoretical framing, leaving the research questions empirically orphaned. Conversely, a theoretically sophisticated paper can be undermined by a poorly evidenced gap. Mastery requires all four.
7.1 Case Study: Energy-Efficient Edge Inference
- Appraisal. Twenty-three primary studies are appraised and tagged by dataset, quantisation method, hardware target and reported metrics. Studies are flagged where latency, energy and accuracy are not reported jointly.
- Synthesis. A thematic synthesis yields three convergent themes: the quantisation-depth versus accuracy trade-off, with most studies converging on 8-bit as a Pareto sweet spot; hardware–software co-design yielding larger gains than algorithmic optimisation alone; and workload heterogeneity, with findings diverging between vision and speech.
- Gap (methodological). Most studies optimise for accuracy under a fixed energy budget; few explore adaptive policies that respond to runtime context such as battery level, network latency or user activity.
- Framing. The study adopts the Markov Decision Process formalism as its theoretical lens, with the policy learned by deep reinforcement learning. State, action and reward are derived from system-level measurements.
Despite extensive research on quantisation and pruning for energy-efficient edge inference, limited attention has been paid to adaptive policies that dynamically trade accuracy for energy based on runtime context. This is consequential because static configurations underutilise the flexibility of modern edge accelerators. Therefore, this study proposes a reinforcement-learning-based adaptive inference framework and evaluates it on a Jetson Nano platform.
8. Common Pitfalls
| Pitfall | Description | Mitigation |
|---|---|---|
| Cataloguing | Listing studies without evaluation | Always append appraisal and synthesis sentences |
| Gap-by-omission | Claiming a gap simply because no paper was found | Use structured searches to demonstrate absence |
| Theory-of-the-month | Adopting a theory because it is fashionable | Justify theory selection with construct alignment |
| Synthesis-by-citation | Dropping names without integrating findings | Write integrative claims and cite evidence for and against them |
| Mixing levels | Mixing empirical findings with opinion | Clearly distinguish evidence from interpretation |
9. Summary
- Critical appraisal evaluates each study's trustworthiness and contribution, using a structured checklist and the four-validity framework.
- Synthesis integrates evidence into new, defensible claims through narrative, thematic, meta-analytic, integrative or framework approaches.
- Gap identification articulates absences precisely, using a taxonomy, a canonical grammar and four diagnostic tests.
- Theoretical framing anchors the research in established theory, generates testable propositions, and guides every methodological decision.
- The techniques are interdependent; a strong literature review chapter is often the single most cited section of a thesis or paper.
10. Exam Preparation
- Internalise the gap-statement grammar. Practise writing five gap statements across different CS sub-fields.
- Memorise one appraisal framework thoroughly. CASP is the most versatile; know its ten questions and apply two to a sample paper.
- Distinguish synthesis from summary. Be prepared to rewrite a summary paragraph as a genuinely synthetic one.
- Connect theory to method. “TAM suggests testing mediation, hence structural equation modelling is appropriate.”
- Use the threats-to-validity vocabulary — internal, external, construct, statistical conclusion.
- Follow the worked-example format: definition, procedure, example, pitfalls.
10.1 Short-Answer Questions
- Differentiate between critical appraisal and synthesis. Give a one-sentence example of each from a CS paper you have read. 3 marks
- List the four validity types in Cook and Campbell's framework and give one CS-specific threat to each. 4 marks
- What is the difference between a theoretical framework and a conceptual framework? When is each appropriate? 3 marks
- Classify each as empirical, theoretical, methodological, contextual, data or inconsistency: (a) no study has evaluated a proposed algorithm on low-resource Indic languages; (b) two systematic reviews report contradictory effect sizes; (c) adoption theory has not been applied to quantum-computing-as-a-service. 3 marks
10.2 Long-Answer Questions
- Select five papers from a recent CS conference and produce an appraisal table using the three-lens model. 10 marks
- Write a thematic synthesis of five studies on a topic of your choice. Include at least one integrative claim that goes beyond any individual study. 10 marks
- Formulate a gap statement for a hypothetical study on automated test-case generation using large language models, and apply the four diagnostic tests. 10 marks
- Propose a theoretical framework for a study on the adoption of AI-assisted code generation tools among undergraduate students. Identify candidate theories, derive two propositions, and justify your choice. 10 marks
10.3 Higher-Order Questions
- A senior researcher argues that “in fast-moving CS sub-fields, formal theoretical framing is impossible because theories become obsolete quickly.” Critically evaluate with reference to at least two theories discussed above.
- Some authors argue that systematic reviews are inappropriate for CS because the field lacks standardised reporting conventions. Defend or refute this position.
- Consider the synthesis claim “Across all surveyed studies, larger models always outperform smaller ones.” Identify the synthesis technique used, the assumptions embedded in the claim, and the conditions under which it would not hold.
11. Further Reading
- Booth, A., Sutton, A., & Papaioannou, D. (2016). Systematic Approaches to a Successful Literature Review. SAGE.
- Cronin, P., Ryan, F., & Coughlan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), 38–43.
- Kitchenham, B., & Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report.
- Müller-Bloch, C., & Kranz, J. (2015). A framework for rigorously identifying research gaps in qualitative literature reviews. ICIS 2015.
- Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8(1), 45.
- Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), xiii–xxiii.
- Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies. EASE 2014.